Outlier detection and management

ABSTRACT

A method, apparatus, system, and computer program products for managing a set of outliers in test data. A computer system analyzes a set of features derived from the test data using different outlier detection methods to generate a result of the set of outliers identified by the different outlier detection methods. The test data is obtained from testing a physical structure. The computer system determines a causality for the set of outliers in the result. The physical structure is retested with a set of changes determined using the causality identified for the set of outliers. The retesting generates new test data for the physical structure.

BACKGROUND INFORMATION 1. Field

The present disclosure relates generally to improved computer system andin particular, to outlier management of test data.

2. Background

In aerospace, automotive, and other industries, physical tests areperformed for products and physical structures forming the products. Thetest data resulting from the test or recorded and process to evaluateperformance, evaluate a quality of design, validate a simulation model,or analyze the data to certify the structure or product. These tests caninclude testing structures such as composite parts. For example, withcomposite parts, physical tests are performed that generate data aboutthe response of the composite parts. The response can be measured usingproperties such as tensile strength, compressive strength, flexuralproperties, shear strength, void content, dynamic mechanical properties,and other properties.

The accuracy of test data generated from performing tests on compositeparts is important to properly evaluate physical structures and to meetgovernment regulations in order to certify the physical structures foruse. Outliers in the test data can affect the validity of the test data.These outliers can include noise. Reducing outliers to meet standardsfor valid test data can be more time-consuming and challenging thandesired.

SUMMARY

In one illustrative example, a method manages a set of outliers in testdata. A computer system analyzes a set of features derived from the testdata using different outlier detection methods to generate a result ofthe set of outliers identified by the different outlier detectionmethods. The test data is obtained from testing a physical structure.The computer system determines a causality for the set of outliers inthe result. The physical structure is retested with a set of changesdetermined using the causality identified for the set of outliers. Theretesting generates new test data for the physical structure. Accordingto other illustrative examples, a system and a computer program productfor managing outliers are provided.

The features and functions can be achieved independently in variousexamples of the present disclosure or may be combined in yet otherexamples in which further details can be seen with reference to thefollowing description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the illustrative examplesare set forth in the appended claims. The illustrative examples,however, as well as a preferred mode of use, further objectives andfeatures thereof, will best be understood by reference to the followingdetailed description of an illustrative example of the presentdisclosure when read in conjunction with the accompanying drawings,wherein:

FIG. 1 is a pictorial representation of a network of data processingsystems in which illustrative examples may be implemented;

FIG. 2 is an illustration of a test data environment in accordance withan illustrative example;

FIG. 3 is an illustration of types of features in accordance with anillustrative example;

FIG. 4 is an illustration of outlier types and causes in accordance withan illustrative example;

FIG. 5 is an illustration of outlier detection methods in accordancewith an illustrative example;

FIG. 6 is an illustration of dataflow in identifying causality forvarious outliers in accordance with an illustrative example;

FIG. 7 is an illustration a flowchart of a process for outlier detectionin accordance with an illustrative example;

FIG. 8 is an illustration of a flowchart of a process for managingoutliers in test data in accordance with an illustrative example;

FIG. 9 is an illustration of a flowchart of a process for detectingoutliers in accordance with an illustrative example;

FIG. 10 is an illustration of a flowchart of a process for featureidentification in accordance with an illustrative example;

FIG. 11 is an illustration of a flowchart of a process for removingnoise in accordance with an illustrative example;

FIG. 12 is an illustration a flowchart of a process for determiningcausality in accordance with an illustrative example;

FIG. 13 is an illustration a flowchart of a process for retesting for aphysical structure in accordance with an illustrative example;

FIG. 14 is an illustration a flowchart of another process for retestinga physical structure in accordance with an illustrative example;

FIG. 15 , an illustration a flowchart of a process for outlier detectionis depicted in accordance with an illustrative example;

FIG. 16 is an illustration of a block diagram of a data processingsystem in accordance with an illustrative example;

FIG. 17 is an illustration of an aircraft manufacturing and servicemethod in accordance with an illustrative example;

FIG. 18 is an illustration of a block diagram of an aircraft in which anillustrative example may be implemented; and

FIG. 19 is an illustration of a block diagram of a product managementsystem in accordance with an illustrative example.

DETAILED DESCRIPTION

The illustrative examples recognize and take into account one or moredifferent considerations as described below. For example, illustrativeexamples recognize and take into account that it would be desirable tohave a method and apparatus that overcome a technical problem withreducing noise and outliers in test data. For example, the illustrativeexamples recognize and take into account that various factors cancontribute to the presence of outliers in test data. For example,improper test sample preparation, excessive processing defects, improperacquisition of test data during testing, and equipment error are someexamples of factors that can cause outliers and noise to be present intest data. Multiple factors can be present leading to the formation ofdifferent types of outliers, affecting the accuracy of the test data.Undetected outliers can lead to improper characterization of materialproperties of physical structures.

Current outlier detection techniques have a number of challenges. Forexample, the illustrative examples recognize and take into accountcurrent outlier detection techniques are unable to detect all types ofoutliers. Further, as the amount of test data decreases, it becomes morechallenging to detect outliers using current outlier detectiontechniques. Additionally, one manner in which test data is examined foroutliers involves operator know-how, engineering judgment, andexperience. This type of technique can introduce error.

Thus, the illustrative examples provide a method, apparatus, system, andcomputer program product for managing outliers. In one illustrativeexample, a set of features derived from the test data is analyzed usinga plurality of outlier detection methods to generate a result ofoutliers identified by the plurality outlier detection methods, whereinthe test data is obtained from testing a physical structure. A causalityfor a set of outliers in the result matrix is determined. Retesting ofthe physical structure the physical structure can be performed with aset of changes determined based on the causality identified for the setof outliers. The retesting generates new test data for the physicalstructure.

With reference now to the figures and, in particular, with reference toFIG. 1 , a pictorial representation of a network of data processingsystems is depicted in which illustrative examples may be implemented.Network data processing system 100 is a network of computers in whichthe illustrative examples may be implemented. Network data processingsystem 100 contains network 102, which is the medium used to providecommunications links between various devices and computers connectedtogether within network data processing system 100. Network 102 mayinclude connections, such as wire, wireless communication links, orfiber optic cables.

In the depicted example, server computer 104 and server computer 106connect to network 102 along with storage unit 108. In addition, clientdevices 110 connect to network 102. As depicted, client devices 110include client computer 112, client computer 114, and client computer116. Client devices 110 can be, for example, computers, workstations, ornetwork computers. In the depicted example, server computer 104 providesinformation, such as boot files, operating system images, andapplications to client devices 110. Further, client devices 110 can alsoinclude other types of client devices such as mobile phone 118, tabletcomputer 120, and smart glasses 122. In this illustrative example,server computer 104, server computer 106, storage unit 108, and clientdevices 110 are network devices that connect to network 102 in whichnetwork 102 is the communications media for these network devices. Someor all of client devices 110 may form an Internet of things (IoT) inwhich these physical devices can connect to network 102 and exchangeinformation with each other over network 102.

Client devices 110 are clients to server computer 104 in this example.Network data processing system 100 may include additional servercomputers, client computers, and other devices not shown. Client devices110 connect to network 102 utilizing at least one of wired, opticalfiber, or wireless connections.

Program instructions located in network data processing system 100 canbe stored on a computer-recordable storage media and downloaded to adata processing system or other device for use. For example, programinstructions can be stored on a computer-recordable storage media onserver computer 104 and downloaded to client devices 110 over network102 for use on client devices 110.

In the depicted example, network data processing system 100 is theInternet with network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers consisting of thousands of commercial, governmental,educational, and other computer systems that route data and messages. Ofcourse, network data processing system 100 also may be implemented usinga number of different types of networks. For example, network 102 can becomprised of at least one of the Internet, an intranet, a local areanetwork (LAN), a metropolitan area network (MAN), or a wide area network(WAN). In this illustrative example, network data processing system 100can be used to provide a cloud computing environment. FIG. 1 is intendedas an example, and not as an architectural limitation for the differentillustrative examples.

As used herein, “a number of” when used with reference to items, meansone or more items. For example, “a number of different types ofnetworks” is one or more different types of networks.

Further, the phrase “at least one of,” when used with a list of items,means different combinations of one or more of the listed items can beused, and only one of each item in the list may be needed. In otherwords, “at least one of” means any combination of items and number ofitems may be used from the list, but not all of the items in the listare required. The item can be a particular object, a thing, or acategory.

For example, without limitation, “at least one of item A, item B, oritem C” may include item A, item A and item B, or item B. This examplealso may include item A, item B, and item C or item B and item C. Ofcourse, any combinations of these items can be present. In someillustrative examples, “at least one of” can be, for example, withoutlimitation, two of item A; one of item B; and ten of item C; four ofitem B and seven of item C; or other suitable combinations.

As depicted, physical structures in the form composite parts 130 aretested in testing facility 132. The testing of composite parts 130 caninclude the determination of bulk properties through the use of tension,compression, and share tests. These tests can be used explore propertiesof composite parts 130 such as open hole tension, open hole compression,interim laminar fracture toughness, compression after impact, fatigue,and other properties. These tests can be performed over a range ofenvironments. This testing of composite parts 130 generates test data134. In this illustrative example, client computer 114 sends test data134 over network 102 to data manager 136 located in server computer 104for processing.

Data manager 136 manages test data 134. For example, data manager 136can analyze test data 134, identify outliers in test data 134, reducenoise in test data 134, or perform other operations on test data 134. Inmanaging test data 134, data manager 136 can identify outliers moreefficiently as compared to current techniques for outlier detection. Forexample, data manager 136 can use multiple ones of outlier detectionmethods 138 to analyze test data 134 to detect outliers 140 in test data134. In this illustrative example, outlier detection methods 138comprises detection methods that are suitable for detecting differenttypes of outliers 140.

The selection of outlier detection methods 138 can be made such thatdifferent outlier detection methods can identify different outliers suchthat the group of outlier detection methods 138 can detect more types ofoutliers as compared to individual outlier detection methods. As result,the selection of two or more of different ones of detection methods 138can provide increase performance in detecting outliers as compared tocurrent techniques.

In this illustrative example, data manager 136 can identify outliertypes 142 for outliers 140 identified in test data 134. Additionally,data manager 136 can identify the cause of outliers 140 based on thedetermination of outlier types 142. By identifying the cause of outliers140, data manager 136 can determine whether the testing of compositeparts 130 is needed.

Data manager 136 can retest composite parts 130 in a number of differentways. The retesting can be initiated by data manager 136 sending atleast one of instructions, commands, or other types of information totesting facility 132. The retesting of composite parts 130 can involvechanging one or more measurement processes used by testing facility 132to test composite parts 130.

As another example, data manager 136 may determine that one or more ofoutliers 140 is caused by composite parts 130 not meeting specificationsfor composite parts 130. With this type of causation of an outlier, datamanager 136 can retest composite parts 130 by having composite parts 130remanufactured with changes in manufacturing such that theremanufactured version of composite parts 130 meet specifications forcomposite parts 130. This retesting generates new test data that can beanalyzed.

Further, in managing test data 134 data manager 136 can also removenoise 143 from test data 134. Thus, data manager 136 can manage testdata 134 to provide test data 134 that can be used to show thatcomposite parts 130 meets criteria or specifications specified byregulations, a manufacturer, or other source.

With reference next to FIG. 2 , an illustration of a test dataenvironment is depicted in accordance with an illustrative example. Inthis illustrative example, test data environment 200 includes componentsthat can be implemented in hardware such as the hardware shown innetwork data processing system 100 in FIG. 1 .

In this illustrative example, outlier management system 202 in test dataenvironment 200 can operate to process test data 204. As depicted, testdata 204 is generated from testing of physical structure 206 by testingsystem 208.

In this illustrative example, the testing of physical structure 206 is aset of physical tests performed on physical structure 206. Physicalstructure 206 can take a number of different forms. For example,physical structure 206 can be selected from a group comprising acomposite part, a test coupon, an assembly, a system, an alloy part,metal structure, and other types of physical structures. In thisillustrative example, a composite part can be, for example, a skinpanel, a wing, or some other suitable type of composite part. A systemor assembly can be comprised of multiple materials. For example, anassembly can be comprised of parts formed from composite materials,plastic materials, and metal materials.

Testing system 208 can be one or more pieces of test equipment that canbe used to perform tests on physical structure 206. This test equipmentcan be used to perform testing such as hardness testing, impact testing,fracture toughness testing, pretesting, fatigue testing, nondestructivetesting, and other types of testing to generate test data 204.

Test data 204 is sent from testing system 208 to outlier managementsystem 202 for processing. As depicted, outlier management system 202comprises data manager 210 in computer system 212.

Data manager 210 can be implemented in software, hardware, firmware or acombination thereof. When software is used, the operations performed bydata manager 210 can be implemented in program code configured to run onhardware, such as a processor unit. When firmware is used, theoperations performed by data manager 210 can be implemented in programcode and data and stored in persistent memory to run on a processorunit. When hardware is employed, the hardware can include circuits thatoperate to perform the operations in data manager 210.

In the illustrative examples, the hardware can take a form selected fromat least one of a circuit system, an integrated circuit, an applicationspecific integrated circuit (ASIC), a programmable logic device, or someother suitable type of hardware configured to perform a number ofoperations. With a programmable logic device, the device can beconfigured to perform the number of operations. The device can bereconfigured at a later time or can be permanently configured to performthe number of operations. Programmable logic devices include, forexample, a programmable logic array, a programmable array logic, a fieldprogrammable logic array, a field programmable gate array, and othersuitable hardware devices. Additionally, the processes can beimplemented in organic components integrated with inorganic componentsand can be comprised entirely of organic components excluding a humanbeing. For example, the processes can be implemented as circuits inorganic semiconductors.

Computer system 212 is a physical hardware system and includes one ormore data processing systems. When more than one data processing systemis present in computer system 212, those data processing systems are incommunication with each other using a communications medium. Thecommunications medium can be a network. The data processing systems canbe selected from at least one of a computer, a server computer, a tabletcomputer, or some other suitable data processing system.

As depicted, computer system 212 includes a number of processor units214 that are capable of executing program instructions 216 implementingprocesses in the illustrative examples. As used herein a processor unitin the number of processor units 214 is a hardware device and iscomprised of hardware circuits such as those on an integrated circuitthat respond and process instructions and program code that operate acomputer. When a number of processor units 214 execute programinstructions 216 for a process, the number of processor units 214 is oneor more processor units that can be on the same computer or on differentcomputers. In other words, the process can be distributed betweenprocessor units on the same or different computers in a computer system.Further, the number of processor units 214 can be of the same type ordifferent type of processor units. For example, a number of processorunits can be selected from at least one of a single core processor, adual-core processor, a multi-processor core, a general-purpose centralprocessing unit (CPU), a graphics processing unit (GPU), a digitalsignal processor (DSP), or some other type of processor unit.

In this illustrative example, data manager 210 in computer system 212can detect outliers 218 for set of features 220 using different outlierdetection methods 222. In this illustrative example, data manager 210can identify the set of features 220 from test data 204. In thisillustrative example, a feature in the set of features 220 represents anindividual measurable property or characteristic of a phenomenon forphysical structure 206. The feature can be a physical response orfailure mode that can occur for physical structure 206. For example, afeature can be a peak in a forced displacement curve, a length of acrack in an image, or some other suitable feature.

Different outlier detection methods 222 can take a number of differentforms. For example, different outlier detection methods 222 can beselected from two or more of a cosine similarity, a correlationanalysis, a principal component analysis, a Sprague-Geers analysis, arobust principal component analysis, or some other suitable outlierdetection method that is currently available. In this example, differentoutlier detection methods 222 can compare different parameters. Forexample, different outlier detection methods can compare at least one oflengths, angles, Eigen values, Eigen vectors, or other metrics orcombinations of metrics used for outlier detection in different outlierdetection methods 222.

Further, the phrase “two or more of,” when used with a list of items,means different combinations of two or more of the listed items can beused, and only one of each item in the list may be needed. In otherwords, “two or more of” means any combination of items and number ofitems may be used from the list, but not all of the items in the listare required. The item can be a particular object, a thing, or acategory.

For example, without limitation, “two or more of item A, item B, or itemC” may include item A and item B, item A and item C, or item B and itemC. This example also may include item A, item B. Of course, anycombinations of these items can be present. In some illustrativeexamples, “two or more of” can be, for example, without limitation, twoof item A and item B; one of item B and 3 of item C; and ten of item C;four of item B and seven of item C; or other suitable combinations.

As depicted, data manager 210 generates result 224 of a set of outliers218 identified by each outlier detection in different outlier detectionmethods 222. In one illustrative example, result 224 can take the formof result matrix 226 for identifying outliers 218. Result matrix 226identifies outliers 218 and outlier detection methods that identifiedoutliers 218.

Further, data manager 210 can determine causality 228 for the set ofoutliers 218. Causality 228 for an outlier in the set of outliers 218 isan identification of a cause of the outlier. Causality 228 can bedetermined using historical data 230 from prior tests and analysis ofphysical structures.

In some cases, and outlier in the set of outliers 218 can be an outlierwithout causality 228. This type of outlier can be discarded.

In determining causality 228, data manager 210 identifies a set ofoutlier types 232 for an outlier in result 224. In other words, anoutlier can have one more than one outlier type. Data manager 210determines causality 228 for the outlier using the set of outlier types232 identified for the outlier.

In this illustrative example, data manager 210 can retest physicalstructure 206 with a set of changes 234 determined using causality 228identified for the set of outliers 218. The set of changes 234 can takea number of different forms. For example, the set of changes 234 can beselected from at least one of changes comprises at least one of ameasurement process change, a geometry change, a manufacturing parameterchange, a manufacturing process change, or some other suitable change.

The retesting generates new test data 236 for physical structure 206. Asdepicted, new test data 236 can be analyzed by data manager 210 todetermine a presence of outliers 218.

The retesting can comprise retesting physical structure 206 using achange to measurement process 238 used in testing system 208 in responseto causality 228 indicating that measurement process 238 was a cause ofan outlier in the set of outliers 218. The change in the measurementprocess can be, for example, changing measurement equipment,recalibrating measurement equipment, changing a measurement technique,adding a new measurement process, changing handling of physicalstructure 206 during testing, or some other suitable change inmeasurement process 238.

In another the example, the retesting can involve data manager 210manufacturing new physical structure 240 with the set of changes 234identified and retesting new physical structure 240. In thisillustrative example, physical structure 206 may have an incorrectdimension. For example, physical structure 206 may have a thickness thatis less than the specified thickness for physical structure 206. In thisexample, the change is a geometry change made to remanufacture isphysical structure 206 with the specified thickness. In another example,physical structure 206 using a waterjet. The waterjet causeinconsistencies. The change can include cutting physical structure 206using a diamond cutter.

Additionally, data manager 210 can remove noise 242 in outliers 218 fromtest data 204 as well as other types of outliers 218. For example, datamanager 210 can remove noise 242 from the set of features 220 using atleast one of removing noise 242 in outliers 218 prior to analyzing theset of features 220 derived from test data 204 using different outlierdetection methods 222 or using an outlier detection method in differentoutlier detection methods 222 that removes noise 242 in outliers 218.

The illustration of test data environment 200 in FIG. 2 is not meant toimply physical or architectural limitations to the manner in which anillustrative example may be implemented. Other components in addition toor in place of the ones illustrated may be used. Some components may beunnecessary. Also, the blocks are presented to illustrate somefunctional components. One or more of these blocks may be combined,divided, or combined and divided into different blocks when implementedin an illustrative example.

For example, the testing of physical structure 206 can be performedusing a simulation of physical structure 206. With this type ofimplementation, testing system 208 can be, for example, a simulationmodel such as a finite element analysis model.

Additionally, the illustrative example can also use different types offeatures including transform features based on theory and governingphysics of the problem to find the underlying features. This type offeature can also be referred to as a theory guided featuretransformation. This transformed data can be analyzed using thedifferent outlier detection methods with greater accuracy as compared tocurrent techniques.

In yet another example, retesting physical structure 206 is unnecessary.Test data 204 with the removal of outliers 218 can provide the qualityneeded for using test data 204. For example, test data 204 with theremoval of outliers 218 can be used validate a simulation model with theoutliers 218 removed from features 220.

In another illustrative example, noise removal 250 can be used to removenoise 242 in test data 204 prior to detecting outliers 218 usingdifferent outlier detection methods 222. For example, noise removal 250can use filters for noise reduction processes that may not detectoutliers 218.

Turning now to FIG. 3 , an illustration of types of features is depictedin accordance with an illustrative example. In the illustrativeexamples, the same reference numeral may be used in more than onefigure. This reuse of a reference numeral in different figuresrepresents the same element in the different figures.

Features 220 can take a number of different forms that can be identifiedby data manager 210 for use in detecting outliers 218 using differentoutlier detection methods 222. As depicted, the set of features 220 canbe selected from at least one of standard feature 300, transformedfeature 302, or selected feature 304. In other words, the set offeatures 220 can be one or more of these feature types and can be in anycombination.

In this illustrative example, standard feature 300 comprises test data204 from a test. In this illustrative example, test data 204 can be, forexample, data from multiple tests performed multiple times or from asingle test performed multiple times. Standard feature 300 can be, forexample, test data for a force displacement curve from values in thetest data from a test measuring displacement of physical structure 206in response to force.

In another example, standard feature 300 can be an image. The image canbe an image of a failure, fracture, the elimination, or other type ofimage showing the response of physical structure 206.

In this illustrative example, transformed feature 302 is test data 204that has been transformed by a mathematical operation. Transformedfeature 302 can also be referred to as a theory guided feature. Forexample, transformed feature 302 can be result of a derivative of testdata 204, an integral of test data 204, or from some other mathematicaloperation performed on test data 204.

For example, the derivative of test data 204 from a force displacementtest can be a tangent or secant stiffness. The integral of test datafrom a force displacement test can be fractured energy. The selection ofthe operation performed in the test data is driven by an understandingof the response of the material. For example, the response can be afailure, delamination, or fracture of the material.

Selected feature 304 is a selection of a portion of test data 204. Forexample, selected feature 304 can be a peak of a force displacementcurve, an area of interest in a force displacement curve, a slope of aline, or other selected feature. In other words, selected feature 304can be a subset of test data 204 containing data of interest.

In another example, selected feature 304 can be a portion of an image.For example, selected feature 304 can be a length of a crack, a numberof parallel lines, or some other selected feature in an image.

In one illustrative example, one or more technical solutions are presentthat overcome a technical problem with detecting outliers in test data.As a result, one or more technical solutions can provide a technicaleffect enabling managing outliers using a plurality of different typesof outlier detection methods.

Turning to FIG. 4 , an illustration of outlier types and causes isdepicted in accordance with an illustrative example. In thisillustrative example, table 500 depicts outlier types 401 and causes 421of outlier types 401. These outlier types are examples of some of thedifferent types of outliers that can be present along with causes 421for outlier types 401. As depicted, outlier types 401 comprise type 1402, type 2 404, type 3 406, type 4 408, type 5 410, type 6 412, andtype 7 414.

In this illustrative example, type 1 402 is caused by premature failure403. Premature failure 403 can be due to defects or other issuesresulting from manufacturing variability in physical structure 206. Type2 404 is caused by stiffness variation 405. In this example, stiffnessvariation 405 can result from, for example, from at least one ofimproper strange gauge measurements, improper layout, or impropercutting of physical structure 206.

As depicted, type 3 406 is caused by stiffness error 407. Stiffnesserror 407 can occur due to tabs sliding, loading fixture rotation, orother test fixture issues in handling the physical structure 206 duringthe testing process.

Type 4 408 is caused by nonlinear variation 409. Nonlinear variation 409can be identified from jumps in loading and load displacement data whichcan occur in response to manufacturing variability in physical structure206.

In another example, type 5 410 is caused by noisy data 411, which can becaused by test equipment. For example, a faulty strain gauge or otherdata acquisition system equipment can cause this type of outlier. Noisefiltering may be used to obtain an acceptable amount of data withoutretesting. Type 6 412 is caused by unexpected load drop 413. Thisunexpected load drop can occur due to delamination of physical structure206.

As depicted, type 7 414 is caused by different damage modes 415. Thesedamage modes are present in images of the physical structure and arecaused by testing the physical structure.

The outlier types are examples of different outlier types for outliers218 in FIG. 2 . By identifying the different types of outliers 218,causes 421 associated with those outliers can be identified. Some ofcauses 421 may simply require performing noise removal.

In other illustrative examples, causes 421 can be true outliers that maybe discarded. In other illustrative examples, causes of causes 421 canbe used to determine a set of changes 234 that may need to be performedin retesting physical structure 206. This retesting may include at leastone of changes to the measurement processor or remanufacturing physicalstructure 206 with changes to obtain suitable test data.

With reference next to FIG. 5 , an illustration of outlier detectionmethods is depicted in accordance with an illustrative example. Table500 identifies the outlier detection method in column 502 and thefeature type in column 504 for the data analyzed by the outlierdetection method.

Table 500 also identifies outlier types that can be detected by eachoutlier detection method using a particular feature type. In thisexample, type 1 is column 508, type 2 is column 510, type 3 is column512, type 4 is column 514, type 5 is column 516, type 6 is column 518,and type 7 is in column 519. These outlier types correspond to theoutlier types 401 described in FIG. 4 .

In row 520, robust principal component analysis (RPCA) is the outlierdetection that that can be used with standard features. This type ofoutlier detection method can compare sparse matrices. This type ofoutlier detection method can also automatically remove noise as part ofthe outlier detection process. With standard features, robust principalcomponent analysis can detect type 1, type 5, and type 6 outliers.

Next, in row 522, Sprague-Geers (SG) comprehensive error is a type ofoutlier detection method that can be used to detect outliers usingstandard features. With this type of test data, Sprague-Geerscomprehensive error can detect type 1, type 2, and type 3 outliers.

Next, robust principal component analysis in row 524 can be used withtransform features to detect outliers. With transform features, robustprincipal component analysis can detect type 1, type 3, type 4, type 5,and type 6 outliers. In row 526, robust principal component analysis isused with selected features to detect outliers. With the use of selectedfeatures, robust principal component analysis can detect type 1, type 2,type 3, and type 4 outliers.

In row 528, Sprague-Geers (SG) comprehensive error is used with selectedfeatures to detect outliers. In this example, Sprague-Geers (SG)comprehensive error can detect type 1, type 4, and type 7 outliers.

Thus, with selection of different outlier detection methods, all of thedifferent types of outliers can be detected. This illustration is notmeant to limit the manner in which other selections of outliers can bemade for different outlier detection methods 222 in FIG. 2 . Othernumbers and other types of outlier detection methods can be useddepending on the particular implementation. The selection of the outlierdetection methods for use in different outlier detection methods 222 canbe made such that the different types of outliers can be detected. Asresult, all outlier types of interest can be detected even though aparticular outlier detection method is unable to detect all of theoutlier types of interest.

Turning to FIG. 6 , an illustration of dataflow in identifying causalityfor various outliers is depicted in accordance with an illustrativeexample. As depicted, historical experimental data 600 is test data 602obtained from prior testing of physical structures. Historicalexperimental data 600 comprises test data 602 from tests performed on aphysical structure. Historical experimental data 600 can be labeled toform labeled historical data 604 (operation 601). In this illustrativeexample, labeled historical data 604 can include test dataidentification 609, outlier presence 606, outlier type 608, and cause610 for the different outlier types.

This information can be determined by engineers or other subject matterexperts analyzing test data 602 from different tests and historicalexperimental data 600. In other illustrative examples, the test data canalso be analyzed using models or other software systems.

In this illustrative example, outlier detection code 612 is program codeimplementing outlier detection methods. Outlier detection code 612 canreceive test data 602 (operation 603) and detect outliers usingdifferent outlier detection methods (operation 605).

The result of this process is analyzed historical data 614. Analyzedhistorical data 614 identifies which of the different outlier detectionmethods identified in columns 617 detected outliers in different sets oftest data 602 identified in rows 615 in analyzed historical data 614.

Labeled historical data 604 is compared with analyzed historical data614 (operation 607), resulting in causality map 620. As depicted,causality map 620 identifies outlier detection methods and what outlierscan be detected by each of the outlier detection method in section 622.Outlier types for the detected outliers in section 622 identified insection 624. Causality for outlier types in section 624 identified insection 626.

In this illustrative example, causality 228 four outliers 218 detectedusing different outlier detection methods 222 in FIG. 2 can beidentified using causality map 620. As result, if the test data is notsufficient or outliers 218 are not truly outliers but have causes, a setof changes 234 can be identified for use in retesting physical structure206.

The illustrative example enables more accurate outlier detection withsmaller amounts of test data as compared to current techniques in whichthe accuracy of outlier detection in these current techniques depend onthe amount of test data available. Further, features in the test datacan be analyzed using a plurality of different outlier detection methodsin the illustrative example.

Thus, the illustrative example can use these different featurescollectively to detect many types of outliers as well as remove noisefrom data. This processing of test data can allow identifying the trueresponse to the material absent with the reduction or removal ofoutliers and noise from the test data.

Computer system 212 can be configured to perform at least one of thesteps, operations, or actions described in the different illustrativeexamples using software, hardware, firmware or a combination thereof. Asa result, computer system 212 with data manager 210 operates as aspecial purpose computer system in which in computer system 212 enablesdetecting outliers and causes of outliers a more efficient manner ascompared to current techniques. For example, data manager 210 transformscomputer system 212 into a special purpose computer system as comparedto currently available general computer systems that do not have datamanager 210.

In the illustrative example, the use of data manager 210 in computersystem 212 integrates processes into a practical application formanaging outliers in test data to meet goals. This managing of outliersincludes retesting a physical structure when outliers are not trueoutliers and have causes that are identified. With the identified of thecauses, data manager 210 in computer system 212 can retest the physicalstructure with a set of changes identified using because is determinedthe outliers.

With reference to FIG. 7 , an illustration of a flowchart of a processfor outlier detection is depicted in accordance with an illustrativeexample. The process illustrated in this figure can be implemented inhardware, software, or both. When implemented in software, the processcan take the form of program code that is run by one of more processorunits located in one or more hardware devices in one or more computersystems. For example, the process can be implemented in data manager 210in computer system 212 in FIG. 2 .

The process begins by testing coupons (operation 700). Coupons areexamples of test structures and can be manufactured using compositematerials in this example. In operation 700, the coupons can havedifferent orientations for layers of composite materials. Further, tabcuts, notches, or other features can also be cut into the coupons fortesting. The testing can include pulling, twisting, bending, andapplication loads to determine the response the coupons to thesedifferent types of forces.

The process generates test data from the testing of test coupons(operation 702). The test data can include, for example, forceddisplacement data images of the coupons and other data.

The process then performs feature engineering to identify features fromthe test data (704). In this example, features can be standard features,transform features, selected features, or some combination thereof. Withcoupons, transform features can be created using a derivative of thetest data can be performed to obtain tangent or secant stiffness. Anintegral of the test data can be performed to determine fractured energyis a transformed feature. Further, different features can be selectedsuch as peak load, sharp drop, cracking, or other features in the testdata or images.

The process then performs outlier detection and noise removal usingdifferent outlier detection methods (operation 706). The result ofoperation 706 identifies the outliers detected by the different outlierdetection methods. Operation 706 can also optionally include removingnoise from the test data before using outlier detection processes. Thenoise removal can be performed using filtering processes. In othercases, noise can be an outlier that can be removed as part of theoutlier detection process.

The process identifies outlier types and causes (operation 708). Theprocess terminates thereafter.

Turning next to FIG. 8 , an illustration of a flowchart of a process formanaging outliers in test data is depicted in accordance with anillustrative example. The process in FIG. 8 can be implemented inhardware, software, or both. When implemented in software, the processcan take the form of program code that is run by one of more processorunits located in one or more hardware devices in one or more computersystems. For example, the process can be implemented in data manager 210in computer system 212 in FIG. 2 .

The process begins by analyzing a set of features derived from the testdata using different outlier detection methods to generate a result of aset of outliers identified by the different outlier detection methods,wherein the test data is obtained from testing a physical structure(operation 800). The process determines a causality for the set ofoutliers in the result (operation 802).

The process retests the physical structure with a set of changesdetermined using the causality identified for the set of outliers,wherein the retesting generates new test data for the physical structure(operation 804). The process terminates thereafter.

Turning now to FIG. 9 , an illustration of a flowchart of a process fordetecting outliers is depicted in accordance with an illustrativeexample. The process illustrated in this figure is an example ofadditional operations that can be performed in the process in FIG. 8 .

The process begins by detecting outliers for the set of features usingthe different outlier detection methods (operation 900). The processgenerates the result of the set of outliers detected by each outlierdetection method in the different outlier detection methods (operation9002). The process terminates thereafter.

With reference to FIG. 10 , an illustration of a flowchart of a processfor feature identification is depicted in accordance with anillustrative example. The operation in this flowchart is an example ofan additional operation that can be performed in the process in FIG. 8 .

The process identifies the set of features from the test data, whereinthe set of features is selected from at least one of a standard feature,a transformed feature, or a selected feature (operation 1000). Theprocess terminates thereafter.

Turning to FIG. 11 , an illustration of a flowchart of a process forremoving noise is depicted in accordance with an illustrative example.The operation in this flowchart is an example of an additional operationthat can be performed with the process in FIG. 8 .

The process removes noise from the set of features using at least one ofremoving the noise prior to analyzing the set of features derived fromthe test data using the different outlier detection methods or using anoutlier detection method in the different outlier detection methods thatremoves the noise (operation 1100). The process terminates thereafter.

In FIG. 12 , an illustration a flowchart of a process for determiningcausality is depicted in accordance with an illustrative example. Theoperations in this flowchart is an example of one implementation foroperation 802 in FIG. 8 .

The process begins by identifying a set of outlier types for an outlierin the result (operation 1200). The process determines the causality forthe outlier using the set of outlier types identified for the outlier(operation 1202). The process terminates thereafter. The identificationof the causality for the outlier using outlier types can be made basedon historical data. For example, previous data collection and analysisof data can determine what causes of different types of outliers.

With reference next to FIG. 13 , an illustration a flowchart of aprocess for retesting for a physical structure is depicted in accordancewith an illustrative example. The process in FIG. 13 is an example ofone implementation for operation 804 in FIG. 8 .

The process retests the physical structure using a change to ameasurement process in response to the causality indicating that themeasurement process was a cause of an outlier in the set of outliers(operation 1300). The process terminates thereafter.

Next in FIG. 14 , an illustration a flowchart of another process forretesting a physical structure is depicted in accordance with anillustrative example. The process in FIG. 14 is an example of oneimplementation for operation 804 in FIG. 8 .

The process begins by manufacturing a new physical structure with theset of changes identified (operation 1400). In operation 1400, the setof changes can comprise at least one of, a measurement process change, ageometry change, a manufacturing parameter change, or a manufacturingprocess change.

The process retests the new physical structure (operation 1402). Theprocess terminates thereafter.

With reference to FIG. 15 , an illustration a flowchart of a process foroutlier detection is depicted in accordance with an illustrativeexample. The process illustrated in this figure can be implemented inhardware, software, or both. When implemented in software, the processcan take the form of program code that is run by one of more processorunits located in one or more hardware devices in one or more computersystems. For example, the process can be implemented in data manager 210in computer system 212 in FIG. 2 .

The process begins by analyzing a set of features derived from the testdata using different outlier detection methods to generate a result of aset of outliers identified by the different outlier detection methods,wherein the test data is obtained from testing a physical structure(operation 1500).

The process removes the outliers from the features (operation 1502). Theprocess validates a simulation model with the outliers removed from theset of features (operation 1504). The process terminates thereafter.

The flowcharts and block diagrams in the different depicted examplesillustrate the architecture, functionality, and operation of somepossible implementations of apparatuses and methods in an illustrativeexample. In this regard, each block in the flowcharts or block diagramscan represent at least one of a module, a segment, a function, or aportion of an operation or step. For example, one or more of the blockscan be implemented as program code, hardware, or a combination of theprogram code and hardware. When implemented in hardware, the hardwarecan, for example, take the form of integrated circuits that aremanufactured or configured to perform one or more operations in theflowcharts or block diagrams. When implemented as a combination ofprogram code and hardware, the implementation may take the form offirmware. Each block in the flowcharts or the block diagrams can beimplemented using special purpose hardware systems that perform thedifferent operations or combinations of special purpose hardware andprogram code run by the special purpose hardware.

In some alternative implementations of an illustrative example, thefunction or functions noted in the blocks may occur out of the ordernoted in the figures. For example, in some cases, two blocks shown insuccession may be performed substantially concurrently, or the blocksmay sometimes be performed in the reverse order, depending upon thefunctionality involved. Also, other blocks may be added in addition tothe illustrated blocks in a flowchart or block diagram.

Turning now to FIG. 16 , an illustration of a block diagram of a dataprocessing system is depicted in accordance with an illustrativeexample. Data processing system 1600 can be used to implement servercomputer 104, server computer 106, client devices 110, in FIG. 1 . Dataprocessing system 1600 can also be used to implement computer system 212in FIG. 2 . In this illustrative example, data processing system 1600includes communications framework 1602, which provides communicationsbetween processor unit 1604, memory 1606, persistent storage 1608,communications unit 1610, input/output (I/O) unit 1612, and display1614. In this example, communications framework 1602 takes the form of abus system.

Processor unit 1604 serves to execute instructions for software that canbe loaded into memory 1606. Processor unit 1604 includes one or moreprocessors. For example, processor unit 1604 can be selected from atleast one of a multicore processor, a central processing unit (CPU), agraphics processing unit (GPU), a physics processing unit (PPU), adigital signal processor (DSP), a network processor, or some othersuitable type of processor. Further, processor unit 1604 can may beimplemented using one or more heterogeneous processor systems in which amain processor is present with secondary processors on a single chip. Asanother illustrative example, processor unit 1604 can be a symmetricmulti-processor system containing multiple processors of the same typeon a single chip.

Memory 1606 and persistent storage 1608 are examples of storage devices1616. A storage device is any piece of hardware that is capable ofstoring information, such as, for example, without limitation, at leastone of data, program code in functional form, or other suitableinformation either on a temporary basis, a permanent basis, or both on atemporary basis and a permanent basis. Storage devices 1616 may also bereferred to as computer-readable storage devices in these illustrativeexamples. Memory 1606, in these examples, can be, for example, arandom-access memory or any other suitable volatile or non-volatilestorage device. Persistent storage 1608 can take various forms,depending on the particular implementation.

For example, persistent storage 1608 may contain one or more componentsor devices. For example, persistent storage 1608 can be a hard drive, asolid-state drive (SSD), a flash memory, a rewritable optical disk, arewritable magnetic tape, or some combination of the above. The mediaused by persistent storage 1608 also can be removable. For example, aremovable hard drive can be used for persistent storage 1608.

Communications unit 1610, in these illustrative examples, provides forcommunications with other data processing systems or devices. In theseillustrative examples, communications unit 1610 is a network interfacecard.

Input/output unit 1612 allows for input and output of data with otherdevices that can be connected to data processing system 1600. Forexample, input/output unit 1612 can provide a connection for user inputthrough at least one of a keyboard, a mouse, or some other suitableinput device. Further, input/output unit 1612 can send output to aprinter. Display 1614 provides a mechanism to display information to auser.

Instructions for at least one of the operating system, applications, orprograms can be located in storage devices 1616, which are incommunication with processor unit 1604 through communications framework1602. The processes of the different examples can be performed byprocessor unit 1604 using computer-implemented instructions, which canbe located in a memory, such as memory 1606.

These instructions are program instructions and are also referred to asprogram code, computer usable program code, or computer-readable programcode that can be read and executed by a processor in processor unit1604. The program code in the different examples can be embodied ondifferent physical or computer-readable storage media, such as memory1606 or persistent storage 1608.

Program code 1618 is located in a functional form on computer-readablemedia 1620 that is selectively removable and can be loaded onto ortransferred to data processing system 1600 for execution by processorunit 1604. Program code 1618 and computer-readable media 1620 formcomputer program product 1622 in these illustrative examples. In theillustrative example, computer-readable media 1620 is computer-readablestorage media 1624.

Computer-readable storage media 1624 is a physical or tangible storagedevice used to store program code 1618 rather than a media thatpropagates or transmits program code 1618. Computer-readable storagemedia 1624, as used herein, is not to be construed as being transitorysignals per se, such as radio waves or other freely propagatingelectromagnetic waves, electromagnetic waves propagating through awaveguide or other transmission media (e.g., light pulses passingthrough a fiber-optic cable), or electrical signals transmitted througha wire.

Alternatively, program code 1618 can be transferred to data processingsystem 1600 using a computer-readable signal media. Thecomputer-readable signal media are signals and can be, for example, apropagated data signal containing program code 1618. For example, thecomputer-readable signal media can be at least one of an electromagneticsignal, an optical signal, or any other suitable type of signal. Thesesignals can be transmitted over connections, such as wirelessconnections, optical fiber cable, coaxial cable, a wire, or any othersuitable type of connection.

Further, as used herein, “computer-readable media 1620” can be singularor plural. For example, program code 1618 can be located incomputer-readable media 1620 in the form of a single storage device orsystem. In another example, program code 1618 can be located incomputer-readable media 1620 that is distributed in multiple dataprocessing systems. In other words, some instructions in program code1618 can be located in one data processing system while otherinstructions in program code 1618 can be located in one data processingsystem. For example, a portion of program code 1618 can be located incomputer-readable media 1620 in a server computer while another portionof program code 1618 can be located in computer-readable media 1620located in a set of client computers.

The different components illustrated for data processing system 1600 arenot meant to provide architectural limitations to the manner in whichdifferent examples can be implemented. In some illustrative examples,one or more of the components may be incorporated in or otherwise form aportion of, another component. For example, memory 1606, or portionsthereof, can be incorporated in processor unit 1604 in some illustrativeexamples. The different illustrative examples can be implemented in adata processing system including components in addition to or in placeof those illustrated for data processing system 1600. Other componentsshown in FIG. 16 can be varied from the illustrative examples shown. Thedifferent examples can be implemented using any hardware device orsystem capable of running program code 1618.

Illustrative examples of the disclosure may be described in the contextof aircraft manufacturing and service method 1700 as shown in FIG. 17and aircraft 1800 as shown in FIG. 18 . Turning first to FIG. 17 , anillustration of an aircraft manufacturing and service method is depictedin accordance with an illustrative example. During pre-production,aircraft manufacturing and service method 1700 may include specificationand design 1702 of aircraft 1800 in FIG. 18 and material procurement1704.

During production, component and subassembly manufacturing 1706 andsystem integration 1708 of aircraft 1800 in FIG. 18 takes place.Thereafter, aircraft 1800 in FIG. 18 can go through certification anddelivery 1710 in order to be placed in service 1712. While in service1712 by a customer, aircraft 1800 in FIG. 18 is scheduled for routinemaintenance and service 1714, which may include modification,reconfiguration, refurbishment, and other maintenance or service.

Each of the processes of aircraft manufacturing and service method 1700may be performed or carried out by a system integrator, a third party,an operator, or some combination thereof. In these examples, theoperator may be a customer. For the purposes of this description, asystem integrator may include, without limitation, any number ofaircraft manufacturers and major-system subcontractors; a third partymay include, without limitation, any number of vendors, subcontractors,and suppliers; and an operator may be an airline, a leasing company, amilitary entity, a service organization, and so on.

With reference now to FIG. 18 , an illustration of an aircraft isdepicted in which an illustrative example may be implemented. In thisexample, aircraft 1800 is produced by aircraft manufacturing and servicemethod 1700 in FIG. 17 and may include airframe 1802 with plurality ofsystems 1804 and interior 1806. Examples of systems 1804 include one ormore of propulsion system 1808, electrical system 1810, hydraulic system1812, and environmental system 1814. Any number of other systems may beincluded. Although an aerospace example is shown, different illustrativeexamples may be applied to other industries, such as the automotiveindustry.

Apparatuses and methods embodied herein may be employed during at leastone of the stages of aircraft manufacturing and service method 1700 inFIG. 17 .

In one illustrative example, components or subassemblies produced incomponent and subassembly manufacturing 1706 in FIG. 17 can befabricated or manufactured in a manner similar to components orsubassemblies produced while aircraft 1800 is in service 1712 in FIG. 17. As yet another illustrative example, one or more apparatus examples,method examples, or a combination thereof can be utilized duringproduction stages, such as component and subassembly manufacturing 1706and system integration 1708 in FIG. 17 . One or more apparatus examples,method examples, or a combination thereof may be utilized while aircraft1800 is in service 1712, during maintenance and service 1714 in FIG. 17, or both. The use of a number of the different illustrative examplesmay substantially expedite the assembly of aircraft 1800, reduce thecost of aircraft 1800, or both expedite the assembly of aircraft 1800and reduce the cost of aircraft 1800.

The use of data manager 210 to identify outliers, causality foroutliers, and perform retesting can reduce the time needed to certifyparts are systems during certification and delivery 1710. Further, theidentification of outliers and causality for outliers can also be usedto improve the quality of components during component and subassemblymanufacturing 1706.

Turning now to FIG. 19 , an illustration of a block diagram of a productmanagement system is depicted in accordance with an illustrativeexample. Product management system 1900 is a physical hardware system.In this illustrative example, product management system 1900 includes atleast one of manufacturing system 1902 or maintenance system 1904.

Manufacturing system 1902 is configured to manufacture products, such asaircraft 1800 in FIG. 18 . As depicted, manufacturing system 1902includes manufacturing equipment 1906. Manufacturing equipment 1906includes at least one of fabrication equipment 1908 or assemblyequipment 1910.

Fabrication equipment 1908 is equipment that used to fabricatecomponents for parts used to form aircraft 1800 in FIG. 18 . Forexample, fabrication equipment 1908 can include machines and tools.These machines and tools can be at least one of a drill, a hydraulicpress, a furnace, an autoclave, a mold, a composite tape laying machine,an automated fiber placement (AFP) machine, a vacuum system, a roboticpick and place system, a flatbed cutting machine, a laser cutter, acomputer numerical control (CNC) cutting machine, a lathe, or othersuitable types of equipment. Fabrication equipment 1908 can be used tofabricate at least one of metal parts, composite parts, semiconductors,circuits, fasteners, ribs, skin panels, spars, antennas, or othersuitable types of parts.

Assembly equipment 1910 is equipment used to assemble parts to formaircraft 1800 in FIG. 18 . In particular, assembly equipment 1910 isused to assemble components and parts to form aircraft 1800 in FIG. 18 .Assembly equipment 1910 also can include machines and tools. Thesemachines and tools may be at least one of a robotic arm, a crawler, afaster installation system, a rail-based drilling system, or a robot.Assembly equipment 1910 can be used to assemble parts such as seats,horizontal stabilizers, wings, engines, engine housings, landing gearsystems, and other parts for aircraft 1800 in FIG. 18 .

In this illustrative example, maintenance system 1904 includesmaintenance equipment 1912. Maintenance equipment 1912 can include anyequipment needed to perform maintenance on aircraft 1800 in FIG. 18 .Maintenance equipment 1912 may include tools for performing differentoperations on parts on aircraft 1800 in FIG. 18 . These operations caninclude at least one of disassembling parts, refurbishing parts,inspecting parts, reworking parts, manufacturing replacement parts, orother operations for performing maintenance on aircraft 1800 in FIG. 18. These operations can be for routine maintenance, inspections,upgrades, refurbishment, or other types of maintenance operations.

In the illustrative example, maintenance equipment 1912 may includeultrasonic inspection devices, x-ray imaging systems, vision systems,drills, crawlers, and other suitable devices. In some cases, maintenanceequipment 1912 can include fabrication equipment 1908, assemblyequipment 1910, or both to produce and assemble parts that needed formaintenance.

Product management system 1900 also includes control system 1914.Control system 1914 is a hardware system and may also include softwareor other types of components. Control system 1914 is configured tocontrol the operation of at least one of manufacturing system 1902 ormaintenance system 1904. In particular, control system 1914 can controlthe operation of at least one of fabrication equipment 1908, assemblyequipment 1910, or maintenance equipment 1912.

The hardware in control system 1914 can be implemented using hardwarethat may include computers, circuits, networks, and other types ofequipment. The control may take the form of direct control ofmanufacturing equipment 1906. For example, robots, computer-controlledmachines, and other equipment can be controlled by control system 1914.In other illustrative examples, control system 1914 can manageoperations performed by human operators 1916 in manufacturing orperforming maintenance on aircraft 1800. For example, control system1914 can assign tasks, provide instructions, display models, or performother operations to manage operations performed by human operators 1916.

In these illustrative examples, data manager 210 from FIG. 2 can beimplemented in control system 1914 to manage at least one of themanufacturing or maintenance of aircraft 1800 in FIG. 18 . For example,data manager 210 can perform analysis of test data obtained during atleast one of manufacturing or maintenance of a product. The test datacan be for the product or parts forming product. This test data can beanalyzed to determine whether outliers are present and the causes ofthose outliers. When the causes of the outliers of current throughmanufacturing or maintenance processes, changes can be made and theproduct can be retested. This retesting can include using changes to ameasurement process or remanufacturing the product or part.

As result, the use of data manager 210 can provide increased quality inthe product. Further, data manager 210 can also provide test data thatcan be used for various certification or compliance processes to meetregulations or manufacturing requirements.

In the different illustrative examples, human operators 1916 can operateor interact with at least one of manufacturing equipment 1906,maintenance equipment 1912, or control system 1914. This interaction canoccur to manufacture aircraft 1800 in FIG. 18 .

Of course, product management system 1900 may be configured to manageother products other than aircraft 1800 in FIG. 18 . Although productmanagement system 1900 has been described with respect to manufacturingin the aerospace industry, product management system 1900 can beconfigured to manage products for other industries. For example, productmanagement system 1900 can be configured to manufacture products for theautomotive industry as well as any other suitable industries.

Some features of the illustrative examples are described in thefollowing clauses. These clauses are examples of features not intendedto limit other illustrative examples.

Clause 1

A method for managing a set of outliers in test data, the methodcomprising:

-   -   analyzing, by a computer system, a set of features derived from        the test data using different outlier detection methods to        generate a result of the set of outliers identified by the        different outlier detection methods, wherein the test data is        obtained from testing a physical structure;    -   determining, by the computer system, a causality for the set of        outliers in the result; and    -   retesting the physical structure with a set of changes        determined using the causality identified for the set of        outliers, wherein the retesting generates new test data for the        physical structure.

Clause 2

The method according to clause 1 further comprising:

-   -   detecting, by the computer system, outliers for the set of        features using the different outlier detection methods; and    -   generating, by the computer system, the result of the set of        outliers detected by each outlier detection method in the        different outlier detection methods.

Clause 3

The method according to one of clauses 1 or 2, wherein determining thecausality for the set of outliers in the result comprises:

-   -   identifying, by the computer system, a set of outlier types for        an outlier in the result; and    -   determining, by the computer system, the causality for the        outlier using the set of outlier types identified for the        outlier.

Clause 4

The method according to one of clauses 1, 2, or 3 further comprising:

-   -   identifying, by the computer system, the set of features from        the test data, wherein the set of features is selected from at        least one of a standard feature, a transformed feature, or a        selected feature.

Clause 5

The method according to one of clauses 1, 2, 3, or 4 wherein thedifferent outlier detection methods are selected from two or more of acosine similarity, a correlation analysis, a principal componentanalysis, a Sprague-Geers analysis, or a robust principal componentanalysis.

Clause 6

The method according to one of clauses according to one of clauses 1, 2,3, 4, or 5, wherein retesting the physical structure with the set ofchanges comprises:

-   -   retesting, by the computer system, the physical structure using        a change to a measurement process in response to the causality        indicating that the measurement process was a cause of an        outlier in the set of outliers.

Clause 7

The method according to one of clauses 1, 2, 3, 4, 5, or 6, whereinretesting the physical structure with the set of changes comprises:

-   -   manufacturing a new physical structure with the set of changes        identified; and    -   retesting the new physical structure.

Clause 8

The method according to clause 7, wherein the set of changes comprisesat least one of, a measurement process change, a geometry change, amanufacturing parameter change, or a manufacturing process change.

Clause 9

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, or 8 furthercomprising:

-   -   removing, by the computer system, noise from the set of features        using at least one of removing the noise prior to analyzing the        set of features derived from the test data using the different        outlier detection methods or using an outlier detection method        in the different outlier detection methods that removes the        noise.

Clause 10

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, or 9,wherein the result is a result matrix identifying outliers and outlierdetection methods identifying the outliers.

Clause 11

The method according to one of clauses 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10,wherein the physical structure is selected from a group comprising acomposite part, a test coupon, an assembly, a system, an alloy part, anda metal structure.

Clause 12

An outlier management system comprising:

-   -   a computer system; and    -   a data manager in the computer system, wherein the data manager        is configured to:

analyze a set of features derived from test data using different outlierdetection methods to generate a result of a set of outliers identifiedby the different outlier detection methods, wherein the test data isobtained from testing a physical structure;

-   -   determine a causality for the set of outliers in the result; and    -   retest the physical structure with a set of changes determined        using the causality identified for the set of outliers, wherein        the retesting generates new test data for the physical        structure.

Clause 13

The outlier management system according to clause 12, wherein the datamanager is configured to:

-   -   detect outliers for the set of features using the different        outlier detection methods; and    -   generate the result of the set of outliers identified by each        outlier detection method in the different outlier detection        methods.

Clause 14

The outlier management system according to one of clauses 12 or 13,wherein in determining the causality for the set of outliers in theresult, the data manager is configured to:

-   -   identify a set of outlier types for an outlier in the result;        and    -   determine the causality for the outlier using the set of outlier        types identified for the outlier.

Clause 15

The outlier management system according to one of clauses 12, 13, or 14,wherein the data manager is configured to:

-   -   identify, by the computer system, the set of features from the        test data, wherein the set of features is selected from at least        one of a standard feature, a transformed feature, or a selected        feature.

Clause 16

The outlier management system according to one of clauses 12, 13, 14, or15, wherein the different outlier detection methods are selected fromtwo or more of a cosine similarity, a correlation analysis, a principalcomponent analysis, a Sprague-Geers analysis, or a robust principalcomponent analysis.

Clause 17

The outlier management system according to one of clauses 12, 13, 14,15, or 16, wherein in retesting the physical structure with the set ofchanges, the data manager is configured to:

retest the physical structure using a change to a measurement process inresponse to the causality indicating that the measurement process was acause of an outlier in the set of outliers.

Clause 18

The outlier management system according to one of clauses 12, 13, 14,15, 16, or 17, wherein in retesting the physical structure with the setof changes, the data manager is configured to:

-   -   manufacture a new physical structure with the set of changes        identified; and        -   retest the new physical structure.

Clause 19

The outlier management system according to clause 18, wherein the set ofchanges comprises at least one of a measurement process change, ageometry change, a manufacturing parameter change, or a manufacturingprocess change.

Clause 20

The outlier management system according to one of clauses 12, 13, 14,15, 16, 17, 18, or 19, wherein data manager is configured to:

-   -   remove noise from the set of features using at least one of        removing the noise prior to analyzing the set of features        derived from the test data using the different outlier detection        methods or using an outlier detection method in the different        outlier detection methods that removes the noise.

Clause 21

The outlier management system according to one of clauses 12, 13, 14,15, 16, 17, 18, 19, or 20, wherein the result is a result matrixidentifying outliers and outlier detection methods identifying theoutliers.

Clause 22

The outlier management system according to one of clauses 12, 13, 14,15, 16, 17, 18, 19, 20, or 21, wherein the physical structure isselected from a group comprising a composite part, an assembly, a testcoupon, a system, an alloy part, and a metal structure.

Clause 23

A method for managing a set of outliers in test data, the methodcomprising:

-   -   analyzing, by a computer system, a set of features derived from        the test data using different outlier detection methods to        generate a result of the set of outliers identified by the        different outlier detection methods, wherein the test data is        obtained from testing a physical structure;    -   removing, by the computer system, the set of outliers from the        features; and    -   validating, by the computer system, a simulation model with the        outlier removed from the set of features.

Clause 24

A computer program product for managing a set of outliers in test data,the computer program product comprising a computer readable storagemedium having program instructions embodied therewith, the programinstructions executable by a computer system to cause the computersystem to perform a method of:

-   -   analyzing, by a computer system, a set of features derived from        the test data using different outlier detection methods to        generate a result of the set of outliers identified by the        different outlier detection methods, wherein the test data is        obtained from testing a physical structure;    -   determining, by the computer system, a causality for the set of        outliers in the result; and    -   retesting the physical structure with a set of changes        determined using the causality identified for the set of        outliers, wherein the retesting generates new test data for the        physical structure.

Thus, a method, apparatus, system, and computer program product formanaging outliers is provided. In one illustrative example, a computersystem analyzes a set of features derived from the test data usingdifferent outlier detection methods to generate a result of a set ofoutliers identified by the different outlier detection methods. The testdata is obtained from testing a physical structure. The computer systemdetermines a causality for the set of outliers in the result. Thephysical structure is retested with a set of changes determined usingthe causality identified for the set of outliers. The retestinggenerates new test data for the physical structure.

The different processes illustrated in the flowcharts can be used toanalyze test data in smaller amounts as compared to currently availableoutlier detection systems. By analyzing the test data using differentoutlier detection methods, the likelihood of more outliers or outliertypes can be identified.

Further, in the different illustrative examples the original test datacan be tested as standard features without changes, selected features inwhich portions of the test data selected or is transformed data. Withtransformed data, the original test data is transformed based on thetheory governing physics of issues or problems identified for underlyingfeatures in physical structures. Different outlier detection methods maydetect outliers using different types of test data selected from atleast one of standard features, transform features, were selectedfeatures. The selected features can also include selected features fromtransformed test data.

As a result, these features in the different illustrative examples canmore efficiently detect more types of outliers as compared to currentsystems. Further, noise can also be removed. As a result, the test dataprocessed using the illustrative example can reveal the underlying trueresponse of a physical structure more easily as compared to currenttechniques.

The description of the different illustrative examples has beenpresented for purposes of illustration and description and is notintended to be exhaustive or limited to the examples in the formdisclosed. The different illustrative examples describe components thatperform actions or operations. In an illustrative example, a componentcan be configured to perform the action or operation described. Forexample, the component can have a configuration or design for astructure that provides the component an ability to perform the actionor operation that is described in the illustrative examples as beingperformed by the component. Further, To the extent that terms“includes”, “including”, “has”, “contains”, and variants thereof areused herein, such terms are intended to be inclusive in a manner similarto the term “comprises” as an open transition word without precludingany additional or other elements.

Many modifications and variations will be apparent to those of ordinaryskill in the art. Further, different illustrative examples may providedifferent features as compared to other desirable examples. The exampleor examples selected are chosen and described in order to best explainthe principles of the examples, the practical application, and to enableothers of ordinary skill in the art to understand the disclosure forvarious examples with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method for managing a set of outliers in testdata, the method comprising: analyzing, by a computer system, a set offeatures derived from the test data using different outlier detectionmethods to generate a result of the set of outliers identified by thedifferent outlier detection methods, wherein the test data is obtainedfrom testing a physical structure; determining, by the computer system,a causality for the set of outliers in the result; and retesting thephysical structure with a set of changes determined using the causalityidentified for the set of outliers, wherein the retesting generates newtest data for the physical structure.
 2. The method of claim 1 furthercomprising: detecting, by the computer system, outliers for the set offeatures using the different outlier detection methods; and generating,by the computer system, the result of the set of outliers detected byeach outlier detection method in the different outlier detectionmethods.
 3. The method of claim 1, wherein determining the causality forthe set of outliers in the result comprises: identifying, by thecomputer system, a set of outlier types for an outlier in the result;and determining, by the computer system, the causality for the outlierusing the set of outlier types identified for the outlier.
 4. The methodof claim 1 further comprising: identifying, by the computer system, theset of features from the test data, wherein the set of features isselected from at least one of a standard feature, a transformed feature,or a selected feature.
 5. The method of claim 1, wherein the differentoutlier detection methods are selected from two or more of a cosinesimilarity, a correlation analysis, a principal component analysis, aSprague-Geers analysis, or a robust principal component analysis.
 6. Themethod of claim 1, wherein retesting the physical structure with the setof changes comprises: retesting, by the computer system, the physicalstructure using a change to a measurement process in response to thecausality indicating that the measurement process was a cause of anoutlier in the set of outliers.
 7. The method of claim 1, whereinretesting the physical structure with the set of changes comprises:manufacturing a new physical structure with the set of changesidentified; and retesting the new physical structure.
 8. The method ofclaim 7, wherein the set of changes comprises at least one of, ameasurement process change, a geometry change, a manufacturing parameterchange, or a manufacturing process change.
 9. The method of claim 1further comprising: removing, by the computer system, noise from the setof features using at least one of removing the noise prior to analyzingthe set of features derived from the test data using the differentoutlier detection methods or using an outlier detection method in thedifferent outlier detection methods that removes the noise.
 10. Themethod of claim 1, wherein the result is a result matrix identifyingoutliers and outlier detection methods identifying the outliers.
 11. Themethod of claim 1, wherein the physical structure is selected from agroup comprising a composite part, a test coupon, an assembly, a system,an alloy part, and a metal structure.
 12. An outlier management systemcomprising: a computer system; and a data manager in the computersystem, wherein the data manager is configured to: analyze a set offeatures derived from test data using different outlier detectionmethods to generate a result of a set of outliers identified by thedifferent outlier detection methods, wherein the test data is obtainedfrom testing a physical structure; determine a causality for the set ofoutliers in the result; and retest the physical structure with a set ofchanges determined using the causality identified for the set ofoutliers, wherein the retesting generates new test data for the physicalstructure.
 13. The outlier management system of claim 12, wherein thedata manager is configured to: detect outliers for the set of featuresusing the different outlier detection methods; and generate the resultof the set of outliers identified by each outlier detection method inthe different outlier detection methods.
 14. The outlier managementsystem of claim 12, wherein in determining the causality for the set ofoutliers in the result, the data manager is configured to: identify aset of outlier types for an outlier in the result; and determine thecausality for the outlier using the set of outlier types identified forthe outlier.
 15. The outlier management system of claim 12, wherein thedata manager is configured to: identify, by the computer system, the setof features from the test data, wherein the set of features is selectedfrom at least one of a standard feature, a transformed feature, or aselected feature.
 16. The outlier management system of claim 12, whereinthe different outlier detection methods are selected from two or more ofa cosine similarity, a correlation analysis, a principal componentanalysis, a Sprague-Geers analysis, or a robust principal componentanalysis.
 17. The outlier management system of claim 12, wherein inretesting the physical structure with the set of changes, the datamanager is configured to: retest the physical structure using a changeto a measurement process in response to the causality indicating thatthe measurement process was a cause of an outlier in the set ofoutliers.
 18. The outlier management system of claim 12, wherein inretesting the physical structure with the set of changes, the datamanager is configured to: manufacture a new physical structure with theset of changes identified; and retest the new physical structure. 19.The outlier management system of claim 18, wherein the set of changescomprises at least one of a measurement process change, a geometrychange, a manufacturing parameter change, or a manufacturing processchange.
 20. The outlier management system of claim 12, wherein the datamanager is configured to: remove noise from the set of features using atleast one of removing the noise prior to analyzing the set of featuresderived from the test data using the different outlier detection methodsor using an outlier detection method in the different outlier detectionmethods that removes the noise.
 21. The outlier management system ofclaim 12, wherein the result is a result matrix identifying outliers andoutlier detection methods identifying the outliers.
 22. The outliermanagement system of claim 12, wherein the physical structure isselected from a group comprising a composite part, an assembly, a testcoupon, a system, an alloy part, and a metal structure.
 23. A method formanaging a set of outliers in test data, the method comprising:analyzing, by a computer system, a set of features derived from the testdata using different outlier detection methods to generate a result ofthe set of outliers identified by the different outlier detectionmethods, wherein the test data is obtained from testing a physicalstructure; removing, by the computer system, the set of outliers fromthe features; and validating, by the computer system, a simulation modelwith the outlier removed from the set of features.